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Agenda 



Time 

7:30 

8:00 


8:15 

8:30 

9:00 

11:15 

12:00 

12:45 

1:30 

2:15 

3:00 


Topic 

Continental Breakfast 

Welcome and Review Process, Panel Chair & NRC 
Staff 


NASA Capability Roadmap Activity 

15.0 Systems Engineering Cost/Risk Analysis Overview 

-Sub-Team Presentations- 

15.1 Systems Engineering 


- Break - 

15.2 Life Cycle Costing 


- Lunch - 

15.3 Risk Management 

15.4 Safety and Reliability Analysis 
Concluding Summary 

- Break - 

Open Discussion 



Speaker 


Vicki Regenie, NASA 
Stephen Cavanaugh, NASA 

Dr. Alan Wilhite, Georgia 
Tech 

Dr. David Bearden, 
Aerospace Corporation 

Theodore Hammer, NASA 
Dr. Homayoon Dezfuli, NASA 
Stephen Cavanaugh, NASA 

NRC Panel 
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SE Capability Roadmap Team 

V 
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Co-Chairs 

NASA: Stephen Cavanaugh, LaRC 
External: Dr. Alan Wilhite, Georgia Tech 



Team Members 

Government Industry Academia 

Dr. Michael Gilbert, LaRC Dr. David Bearden, Aerospace Dr. Alan Wilhite, Georgia Tech 

Theodore Hammer, HQ Dr. Leonard Brownlow, Aerospace 

Dr. Homayoon Dezfuli, HQ Gaspare Maggio, SAIC 

Stephen Creech, MSFC Steven Froncillo, SAIC 

Phil Napala, HQ 

CAPT Daven Madsen, Navy/NSSO 

Dr. Steve Meier, NRO 

Richard Westermeyer, Navy/NSSO 


Consultants 
Stephen Kapurch, HQ 
David Graham, HQ 
Dale Thomas, MSFC 
Stephen Prusha, JPL 
Chuck Wiesbin, JPL 
Ron Moyer, HQ 


Coordinators 

Directorate: Vicky Hwa, HQ Technical 

Doug Craig, HQ Integration 
Betsy Park, HQ Integration 
APIO: Victoria Regenie, DFRC 
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Capability Definitions 



MV 



* Systems engineering is a robust approach to see to it that 
the system is designed, built, and operated so that it 
accomplishes its purpose in the most cost-effective way 
possible, considering performance, cost, schedule, and risk. 


• Life-Cycle Cost is an integrated, process-centered, and 
disciplined approach to life cycle management of projects 
providing real and tangible benefits to all project 
stakeholders. 


• Risk Management identifies potential problem areas early 
enough to allow development and implementation of 
mitigation strategies to control cost, schedule and mission 
success. 


Safety and Reliability Analysis maximizes Mission Success 
while managing safety risk and affordably meeting mission 
objectives. 





IA 


Capability Roadmap Breakdown 

Structure 



Systems 

Engineering 

15.1 


SE, Cost & 
Risk Analysis 

15.0 


11 duanced P tanning 6 Integration Ij^ffice 

NASA Chair: Steve Cavanaugh (LaRC) 

External Chair: Dr. Alan Wilhite (Georgia Tech) 


Life Cycle 


Risk 


Safety & 

Costing 


Management 


Reliability 

15.2 


15.3 


i5.4 Analysis 


Lead: Dr. Alan Wilhite/GT Lead: Dr. Dave Bearden Lead: Ted Hammer/HQ Lead: Dr. Homayoon Dezfuli/HQ 
/Aerospace 








Engineering 

15.1.1 

Tools 

15.2.1 

Prepare for 
i Risk Management 

15.3.1 

_ System Safety 

15.4.1 


Support 

15.1.2 

Skills 

15.2.2 

Identify & 

- Analyze Risks 

15.3.2 

_ System Reliability 

15.4.2 

Process 
" Management 

15.1.3 

Process 

15.2.3 

Mitigate 

Risks 

15.3.3 

Safety 

L Management 

15.4.3 

Project 

Management 

15.1.4 



This Capability Roadmap scope does not include performing the integration of all 
fifteen Capability Roadmaps. Roadmap coordinators (MD, Center, & APIO) 
comprise the Integration Team and facilitate the integration process by capturing 
Roadmap data and dependencies and documenting in relational database tool. 














Need for Systems Engineering 

V 


• The President has challenged NASA to undertake 
exploration of the solar system 



• In the face of tight budgets and mission risks, it is critical 
that these missions be executed flawlessly 


- Requires sound approach to Systems Engineering 

• Tools, methods, processes 

- Continuous improvement 

• Best of industry and government 

• Standard processes 

- All centers 

- All missions 

- All programs/projects 

• System Engineering must be a “value added proposition” 
not an overhead burden 


- Consistent with the spjrit of_CAIB Recommendation 
NASA’s new vision requires, more than ever, excellence in an 
integrated systems engineering cost/risk analysis capability 
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Four Systems Engineering 
Essentials 



MV 



1 st - Processes & Concepts 

(What) 

A 



4th - How well organization 
implements and supports the 
framework with: 

• Policies & Procedures 

• Process Improvement 

• Human Resources 

• Training 

• Milestone & Decision Gate 
Review Criteria 

• Management of Quality 


►2 nd - Performance Aids 

(How) 


3 rd - Workforce 

(Who) 




% Complexity is a Major Issue 

V 


* Systems-of-Systems are Complex 




As More Systems Are Added, the 
Interfaces Grow in a Non-Linear 
Fashion 

Many of the Existing Systems Are 
Old and Not Built for These Interfaces 

Conflicting or Missing Interface Standards 
Make It Hard to Define Interface Interactions 


* Systems Engineering Must Deal With This Complexity 


- End-to-End Systems Engineering Is Needed, Including 
“Reengineering” Of Old Systems 

- Robust M&S, Verification And Validation Testing Are A Must 

- Need To Upgrade Modeling And Simulation Tools For Both 
Concept Definition And Verification And Validation Phases 

1 Reference: 23 Feb. 2005 - James R van Gaasbeek 

Northrop Grumman Integrated Systems 
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Project Constellation Timeline 



09 i 10 i 11 i 12 . 13 i 14 . 15 i 16 i 17 . 18 i 19 i 20 


CEV Init Fit 
1 st Launch Lunar 
Robotic Orbiter 


1 st Unmanned 
CEV Fit 


• 1 st Crewed • 1 st Human 
CEV Fit Moon Mission 



IOC 


Concept 

Technology 

System Development and Demonstration 

Production & 

Operations 

% Refinement 

Development 

System 

rw 9 " 

System 

Deployment 

&Sipport 

Concept 

Decision 


Integration 

Review 

Demonstration 

OT&E ^ Decision 



A 




Spiral 1 

Crewed Space 
Flight in LEO 


Spiral 2 


KX 


Concept 
<^> Refinement 

Concept 
Decision 


Technology System Development and Denonstrati on Production & Operations 
opment System Design system Deployment & Support 

Integration "ST Deironstration ^ 


♦ 


Tech Maturation for 
Spiral Insertion 


Critical Milestones 
during System Integration 
and Demonstration 
(Notional Only) 




Moon Ext. 
Duration (2015- 
2020) 


Spiral nth? 


Non-advocacy Reviews 
Independent Cost Reviews 








Why is this Capability i 


a 



MV 




September 21, 2004 Letter from the National Academies 



Dear RADM Steidle: 

At your request, the National Research Council recently established the 
Committee on Systems Integration for Project Constellation. 

The following quotes were taken from the report: 


“Strengthening the state of systems engineering is also critical to the long- 
term success of Project Constellation. A competent systems engineering 
capability must be resident within the government and industry”. 


“NASA’s human spaceflight systems engineering capability has eroded 
significantly as a result of declining engineering and development work, 
which has been replaced by operational responsibilities” . 


“The demand for experienced systems engineers, who can function credibly in 

a svstem-of-svstems environment, is particularly acute” . 


“ Plans should, be developed for maintaining a satisfactory base of systems 
engineering throughout the duration of this program ” 
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NASA SE&I Strategy Team 

Preliminary Findings & Analysis 

♦ “Systems Integration” Will Take Place At Multiple Tiers 

• Tiers structured around functional responsibilities 

• Must be prepared to support with maximum efficiency, minimum bureaucracy 

• Need to support Directorate and Technology Themes, as well as Constellation 

• SE&I authority should reside at lowest possible level 

♦ System-of-Systems Integration Demands Creative Solution 

• No single model evaluated by NRC offers complete solution 

• Complete expertise and competence is not available in any one sector 

• Certain functions can only be executed by government personnel 

• “Hybrid model” using government, FFRDC, and industry is attractive 

♦ ESMD SE&I Capability Will Be Phased-ln Over Time 

® Government will perform SE&I work needed to complete Spiral 1 SRR 

• Near-term solution may evolve to different Long-term solution 
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Basis for Assessment 



• Quality Function Deployment (QFD) 

A quality system that implements elements of Systems 
Thinking (viewing the development process as a system) and 
Psychology (understanding customer needs 

• Benchmarking - Chief Engineers Fort Belvoir Workshop on 
February 22-23, 2005 

Learning from the experience of others in Industry, DoD, and 
Other Agencies 

• Literature Search - mostly Internet 


• Limitations of Assessment 

- Budget limitations keep team small and limited in scope 

QFD assessment limited to team size - small sample of NASA 
Assessment more Qualitative vs. Quantitative 
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Capability Readiness Rating for 
process, tools, and skills 


Team Gap Assessment 


APIO Capability Readiness Levefs 



tanning 6 I ntegration U ffice 
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Capability Team 15: Systems Engineering 
Top Level Capability Roadmap 


Key Assumptions: 
Exploration & 
Science 



Capability Roadmap 
15: Systems 
Engineering Risk/Cost 


2008 CEV 
Initial Flight 

a ► 


Initial Prod 
Managers 


Analysis 


jet Life-Cycle 
nt Capability 


2011 James 2013 Comet Surface 2015 Prepare for 

Webb Sample Return Lunar Missions 

*> ► 


Integrated System 
Engineering and 
Management Capability 



Collaborative/Distributive 
PLEM Simulation-Based 
Capability 


15.1 Systems Engineering 


4 e . 

erltatic 


Initial ! 
Implementation 


Validated, Integrated 
SE Environment 


Collaborative/Distributive PLEM 
Simulation-Based Capability 


15.2 Life Cycle Cost 


Agency-wide LCC 
Models & Process 


Continuous Cos 
Risk Management 


Integrated Life Cycle 
Cost Models 


Life Cycle Cost linked to 
Project Management 



15.3 Risk Management 


Integrated Risk Tools and 
Mitigation plans 


Standardized Simulation 
Based Risk Analysis 


Probabilistic Risk Analysis 
and Mitigation 



15.4 Safety & Reliability 


Standard Safety/Reliability Advanced Physics- 

Management Process with SF based QRA 

Process 


Next-generation Hazard 
Analysis Techniques 



♦ 

A 

t 



Major Event / Accomplishment / Milestone 
Ready to Use 


PLM - Product Life Cycle Management 
SBM - Simulation Based Modeling 
CMMI - Capability Maturity Model Integration 
QRA - Quantitative Risk Assessment 
LCC - Life Cycle Cost 
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Capability Team 15: Systems Engineering Top 
Level Capability Roadmap 


Key Assumptions: 2018 Deep Drill & Completed 

Exploration & Initial Human Landing 

Science 


2025 Extended Lunar Capability 
& Life Finder Telescope 








Capability Roadmap 
.... 15: Systems 

Engineering Risk/Cost 
Analysis 

i 

r i 

— *1 

International Collaborative^^ 
Engineering / Management 
Simulation Based Capability^ 

y 






15.1 Systems Engineering 


Initial International Collaborat 
Management Simulation Bas 

ve Engineering/ 
sd Capability 








15.2 Life Cycle Cost 

Decisions based on 
Economic LCC Models 

LCC imbedded in all 
Agency Decisions 









15.3 Risk Management 

Interdependent Systems 
Risk Analysis 

Interactive Risk Identificat 
and Mitigation 

on 

Accurate Risk Analysis in 
Uncertain Environments 





15.4 Safety & Reliability 

Safety & Reliability Informed 
Decision making 

Virtual Safety and SE 
Analysis capability 







▲ 

t 


Major Event / Accomplishment / Milestone 
Ready to Use 


PLM - Product Life Cycle Management 
SBM - Simulation Based Modeling 
CMMI - Capability Maturity Model Integration 
QRA - Quantitative Risk Assessment 
LCC - Life Cycle Cost 
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Future State Required to Meet 
NASA Exploration Vision 



• Process (What) - Need a common process for Systems 

Engineering, Cost, Risk and Safety. NASA Policy 
Requirements, guidelines and handbooks for this Capability 
need to be developed along with a need for an audible 
process. 


• Tools (How) - Need a standardized approach for Systems 

Analysis. This includes a framework for advanced tools. 

* People (Who) - Need qualified personnel. Training & Education 
programs including certification tied to job criteria and 
performance standards. 

“An immediate transformation imperative for all programs 
is to focus more attention on the application of Systems 
Engineering principles and practices throughout the 

system life cycle ” 

USAF Chief of Acquisition Memo, “Incentivizing Contractors for Better 

Systems Engineering, 9 Apr 03 
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Dr. Alan Wilhite 
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Benefits of Systems Engineering 

V 

L A 



• Requirements driven - build the right system 

• Process driven - build the system right 

• Integrated engineering and management 
for informed decisions 

• Less cost / Less duration 
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^ Systems Engineering and Integration 



Systems Management 


Requirements, 
Requirements flowdown, 
Interfaces and Integration 
Performance, 
Specifications, 
Verification and Validation 
Technical Risk, 

Cost 

Engineering 
and Analysis 



Technology Selection, 
Performance trades 


Planning, 

Development, 

Production, 

Operations, 

Decision Analysis and Criteria, 
Cost and Schedule Risk 


Prog ram/Project 
Management 


Requirements trades 
Min Performance Criteria, 
Investment Strategy, 
Best System, 

Verification and Validation, 
Technical Risk 


Ref. GaTech AE 6322 
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The Systems Engineering Process 

(Ref. Mil STD 499B) 



Process Input 

• Customer Needs/Objectives/ 

Requirements 

- Missions 

- Measures of Effectiveness 

- Environments 

- Constraints 

• Technology Base 

• Output Requirements from Prior 
Development Effort 

• Program Decision Requirements/ 

• Requirements Applied Through 
Specifications and Standards 


n 
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Requirements Analysis 

■ Analyze Missions & Environments 

■ Identify Functional Requirements 

■ Define/Refine Performance & Design 
Constraint Requirement 



System Analysis 
& Control 
(Balance) 


Requirement Loop 

Functional Analysis/Allocation 

• Decompose to Lower-Level Functions 

• Allocate Performance & Other Limiting Requirements to 
All Functional Levels 

• Define/Refine Functional Interfaces (Internal/External) 

• Define/Refine/Integrate Functional Architecture 


Trade-Off Studies 
Effectiveness Analysis 
Risk Management 
Configuration Management 
Interface Management 
Performance Measurement 

- SEMS 
-TPM 

- Technical Reviews 


Design Loop 
Synthesis 

• Transform Architectures (Functional to Physical) 

• Define Alternative System Concepts, Configuration 
Items & System Elements 

• Select Preferred Product & Process Solutions 

• Define/Refine Physical Interfaces (Internal/External) 


Related Terms: 

Customer = Organization responsible for Primary Functions 
Primary Functions = Development, Production/Construction, Verification, 
Deployment, Operations, Support Training, Disposal 
Systems Elements = Hardware, Software, Personnel, Facilities, Data, Material, 
Services, Techniques 


Process Output 

Development Level Dependant 

- Decision Data Base 

- System/Configuration Item 
Architecture 

- Specification & Baseline 
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Scope of SE Standards 






CMMI/SE 


IEEE 1220 Application & 
Management of the SE Process 

ANSI/EIA 632 

Processes for Engineering Systems 

ISO/I EC 15288 

System Life Cycle Processes 



MIL-STD-499B * 
Systems Engineering 




Breadth of Scope 



* Mil-Std-499C has more detail (similar to 15288) than Mil-Std 
499B and has more breadth (similar to IEEE 1220) 
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Capability Maturity Model Integration 

V 

L A 


CMMI - DoD developed integrated model for systems engineering, 

software engineering, integrated product process 
development, and supplier sourcing 



duanced 


P tanning 6 


I ntegration 


I Hire 



• Organizational Process Focus 

• Organizational Process 
Definition 

• Organizational Training 

• Organizational Process 
Performance 

• Organizational Innovation 
and Deployment 


• Project Planning 

• Project Monitoring and 
Control 

• Supplier Agreement Mgmt. 

• Integrated Project Mgmt. 

• Integrated Supplier Management 

• Risk Management 

• Quantitative Project Mgmt. 

• Integrated Teaming 


• Requirements Management 

• Requirements Development 

• Technical Solution 

• Product Integration 

• Verification 

• Validation 


• Configuration Mgmt. 

• Process and Product 

• Quality Assurance 

• Measurement & 
Analysis 

• Decision Analysis and 
Resolution 

• Causal Analysis and 
Resolution 

• Organizational 
Environment for 
Integration 


CMMI used as initial basis for strategic planning 
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Overview of the "State” 

V 

L A 



• The Standish Group (which exists solely to track IT successes 
and failures) surveyed 13,522 projects in 2003 and showed the 
following: 


- 34% of projects succeed (these projects are defined as those 
which deliver the contracted capabilities on time and on budget). 

- 15% of projects are out and out failures (these projects are defined 
as those abandoned midstream) 

- The rest (51%) are "challenged", meaning over budget, and/or over 
schedule, and/or deliver less capability / functionality than agreed 
upon and contracted for. 

• According to a Lake & Sheard paper 


- Systems Engineering is practiced in a quagmire of SE Standards 

• MARC Proceedings 1999 

According to the AF Center for Systems Engineering: 

- “Systems Engineering is not broken." 


Ref: Lake Briefing at February 
2005 Ft Belvoir NASA Chief 
Engineer Workshop 


Systems Engineering is not broken but needs significant advancement 

to improve NASA’s program success rate 
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System Engineering 





SE Capability Team Assessment 



SE-CMMI 


Team 

Assessment 


ENGINEERING 



REQUIREMENTS DEVELOPMENT ! 



REQUIREMENTS MANAGEMENT [ 




TECHNICAL SOLUTION 


PRODUCT INTEGRATION 


VERIFICATION 


VALIDATION 


PROJECT MANAGEMENT 



PROJECT PLANNING | 



PROJECT MONITORING AND CONTROL I 




SUPPLIER AGREEMENT MANAGEMENT 




INTEGRATED PROJECT MANAGEMENT FOR IPPD [ 




RISK MANAGEMENT i 




INTEGRATED TEAMING 1 




INTEGRATED SUPPLIER MANAGEMENT 


QUANTITATIVE PROJECT MANAGEMENT 


SUPPORT 



CONFIGURATION MANAGEMENT 



PROCESS AND PRODUCT QUALITY ASSURANCE 


MEASUREMENT AND ANALYSIS 




DECISION ANALYSIS AND RESOLUTION 




ORGANIZATIONAL ENVIRONMENT FOR INTEGRATION 


CAUSAL ANALYSIS AND RESOLUTION 


PROC 

ESS MANAGEMENT 



ORGANIZATIONAL PROCESS FOCUS 


ORGANIZATIONAL PROCESS DEFINITION 


ORGANIZATIONAL TRAINING 


ORGANIZATIONAL PROCESS PERFORMANCE 


ORGANIZATIONAL INNOVATION AND DEPLOYMENT 




Integrated rollup 
of Importance and 
Present Capability 


Critical Gap 1 


Significant Gap 


No or Minor Gap 
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Detail of Capability Assessment 

(Top 10% out of 187 processes) 



Establish Evaluation Criteria 
Identify and Analyze Risks 
Select Solutions 
Evaluate Alternatives 
INTEGRATED TEAMING 
Manage Corrective Action to Closure 
Establish Estimates 
Identify Alternative Solutions 
Objectively Evaluate Work Products and Services 
Evaluate Assembled Product Components 
Obtain an Understanding of Requirements 
ORGANIZATIONAL TRAINING 
Balance Team and Home Organization Responsibilities 
Identify Inconsistencies between Project Work and Requirements 
ORGANIZ ATONAL INNOVATDN AND DEPLOYMENT 
Establish Incentives for Integration 
Establish the Organization’s Shared Vision 
Establish Guidelines for Decision Analysis 



ITice 


0 10 20 30 40 50 60 70 80 


90 






Other Identified SE Capability Gaps 



Systems of Systems Integration 


Experienced SE Personnel 

Standard Process/Process Improvement 

Facilitate Advanced Technology 


Estimate and Manage Costs 


Acquisition Strategy 


Advanced Collaborative Environment 



Refs. 

- NRC SE&I Study, 2004 

- NASA SE Workshop, 2005 


Critical Gap 1 


Significant Gap 


No or Minor Gap 
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Quagmire of SE Standards 


But SE standard writers can’t agree on what should be in a 
Standard - Hence a quagmire! 




ISO/IEC 


ISO/IEC 

12207:1995 


15504:2003 




MIL-STD- 

499C:200? 

Draft 



V IEEE 1220: 


\ 1998 


Systems 

Engineering 

Guide:?00? 


EIA/IS- 

731:1999 

SE-CM 

NDIA/SEI: 
CMMI SW 
20 ( 


NDIA/SEI: 
CMMI SE 
2002 
Verl.1 


ISO/I EC 
24748:200? 
15288/12207 


f Jan 1994 / 

EIA- 


/ > 

632:1999 



\ ^ 




NAVAIR 

\ L 


EIA- 

632:200? 

Update 


IEEE 1220: 
2005 


ISO/IEC/IEEE 
1220:200? 
(Fast Track) 


NDIA/SEI: 
CMMI SE 
200? 
Ver 2 
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Scope of SE Standards 



CMMI/SE 


IEEE 1220 Application & 
Management of the SE Process 

ANSI/EIA 632 

Processes for Engineering Systems 

ISO/I EC 15288 

System Life Cycle Processes 



MIL-STD-499B * 
Systems Engineering 




Breadth of Scope 

* Mil-Std-499C has more detail (similar to 15288) than Mil-Std 
499B and has more breadth (similar to IEEE 1220) 



Ref: Lake Briefing at February 2005 Ft Belvoir NASA Chief Engineer Workshop 
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^ CMMI Recommended Maturation Path 



ML 

CL1 | CL2 

CL3 

CL4 | CL5 1 

1 learn 

1 Assessment 

REQUIREMENTS MANAGEMENT 

2 





MEASUREMENT AND ANALYSIS 

2 





PROJECT MONITORING AND CONTROL 

2 

Maturity 




PROJECT PLANNING 

2 

Level 




PROCESS AND PRODUCT QUALITY ASSURANCE 

2 

2 




SUPPLIER AGREEMENT MANAGEMENT 

2 





CONFIGURATION MANAGEMENT 

2 





DECISION ANALYSIS AND RESOLUTION 

3 




PRODUCT INTEGRATION 

3 




REQUIREMENTS DEVELOPMENT 

3 




TECHNICAL SOLUTION 

3 

Maturity 



VALIDATION 

3 

Level 3 



VERIFICATION 

3 




ORGANIZATIONAL PROCESS DEFINITION 

3 




ORGANIZATIONAL PROCESS FOCUS 

3 




INTEGRATED PROJECT MANAGEMENT FOR IPPD 

3 




RISK MANAGEMENT 

3 




INTEGRATED SUPPLIER MANAGEMENT 

3 




ORGANIZATIONAL TRAINING 

3 




INTEGRATED TEAMING 

3 




ORGANIZATIONAL ENVIRONMENT FOR INTEGRATE 

3 




ORGANIZATIONAL PROCESS PERFORMANCE 

4 

Maturity 



QUANTITATIVE PROJECT MANAGEMENT 

4 

Level 4 



ORGANIZATIONAL INNOVATION AND DEPLOYMENT 

5 

Maturity 



CAUSAL ANALYSIS AND RESOLUTION 

5 

Level 5 




SE Gap Assessment indicates that CMMI Maturity Levels 2 and 3 
should be developed in parallel for NASA 
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Systems Engineering Support to 
Program Management 



Program Management 

Planning 
Organizing 
Directing 
Monitoring 
- Maintaining 
insight or assessing 
Control 



Common 
Areas 

Risk Mgt 
Config Mgt 
Performance 

Assurance 



Systems Engineering 

Requirements & 
Functional Analysis 
and Allocation 
Design Synthesis 
Systems Analysis & 
Control (Balance) 
Integration & 

• Verification 


SE Gap Assessment also agrees with CMMI that Systems Engineering and 
Program Management must be integrated for NASA 
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Enterprise Systems versus 
Program Systems Engineering 





Single Systems 



Enterprise Systems 



Engineering 
(Stand Alone Syst 


Optimal 

Solution 

Space 


End state well define 
Engineered and dev 
within a fixed budge 
and cost 

Well known schedul 
technical, and benef 
baseline 

Often replaces a “le< 

System 

Priority often 

- Technical/Secur 

- Operational 

- Cost 

- Political 


Dynamic end state 
Systems-of-Systems 
evolves over time 
Subject to annual 
budget revisions 
Facilitates Senior 
Decision Makers 
Priority often 

- Political 

- Cost 

- Operational 

- Security 

- Technical 


Engineering 

System-of-Systems) 


Competing Forces Addressed by Systems Engineering 
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Recommended NASA 
SE Process Development 



• Tier 1 : SE Agency Policy and Process Improvement 
Processes 

- Process application policy 

- Architecture, Base and General Processes 

- Knowledge Management and Continuous Process 
Improvement 

• Tier 2: Process Area Procedures 

- Specific standards and references identified 

- Process interfaces (HQ-Center, HQ-Contractor, 

Center-Contractor) 

- System of Systems integration 

- Can be tailored to specific directorate 

• Tier 3: Detailed Guidebooks 

Best practices of how to implement SE 
General tools and methods 

• Tier 4: System Engineering Management Plans 

- Technical program 

- Specific plans on SE implementation 

- Engineering specialty integration 

- Specific tools and methods selected 

- Organizational and contract interfaces defined 




Tiers 3 
Guidebooks 


I 


Tier 4 - Project 
SE Plans for implementation 
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System Engineering Processes 
Assessment and Vision 



Typical Today 

5-Year Vision 

10-year Vision 

15-Year Vision 

• national standard processes exist but 
in a quagmire of interfaces 

• NASA has a SE guideline (NASA 

SP-6105) that is only sporadically 
followed 

• no NASA-wide policy on systems 
engineering exists 

• NASA, DoD, and contractor teams 

use different processes and 
terminology 

• A systems engineering policy, 

guidelines, and implementation 
strategies based on national 
standards and 
NASA/DoD/contractor best 
practices has been developed 

• Annual audits of NASA’s systems 

engineering process model 
ensures best practices are used 
and distributed 

• A systems engineering certification 

program requiring continual 
education and training has been 
institutionalized 

• A knowledge management system 

for capturing and reuse of best 
practices and knowledge 
repository for cost, reliability, 
validated systems analyses and 
simulations, software, and 
hardware has been initiated 

• A completely digital product life- 
cycle management system for 
systems engineering and 
management for program/project 
control has been developed 

• A collaborative / distributive 
advanced engineering 
environment for product life-cycle 
engineering and management has 
been developed based on system 
engineer and management 
processes for systems 
development and workforce 
training 

• Systems engineering, life-cycle 

cost, risk, and safety have been 
integrated for robust solutions of 
complex systems-of-systems 
development 

• All NASA centers have achieved 

the top level of systems 
engineering maturity 

• A certified (educated, trained, and 
experienced) systems engineering 
staff exists for engineering, 
management, and decision 
making 

• the organization interfaces and 

throughput is optimized through 
dynamic simulations 

• an expert system for systems 

engineering exists to aid in the 
training and use of the validated 
advanced engineering 
environment for complex systems- 
of-systems developments 

• Knowledge management has 

revolutionized the startup of new 
programs with reuse of processes 
and tools 

• All decisions are based on validated 

simulations and virtual and 
surgical physical testing for 
performance, cost, safety, 
uncertainty, and risk (and 
politics!!) 

• a completed integrated international 
organization is optimized for the 
collaborative distributed 
environment 
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Systems Engineering 
Architect/Specialist 



• Definition of a Systems Engineering Architect/Expert 

- Architect network centric and systems of systems 

- System Integrator 

- Drives next generation of mission solutions 

• Attributes 

- Experienced technical leader 

- Experienced in working with the customer, understand their 
needs and customer value and to serve as the customer’s 
primary technical interface 

- Expert in fundamentals - cost, schedule, risk, processes 

- System lifecycle experience from pre-proposal to logistics 
support 

- Understand hardware, software, mission and big picture 

- Solid interpersonal skills, verbal and written communications 

• Lack of senior level experienced systems 
eng i neers/arch i tects 

- Many self-proclaimed systems engineers 

- Exists both in industry and government 
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US R&D Scientists and Engineers 
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Degreed workforce is a shrinking pool. 


38 










The Resource Picture 


Degreed workforce is a shrinking pool 



- Many graduates are not US citizens 

- Total engineering enrollments continue to decrease 

20-30 year cycle between major system developments 
and 1 0 year development cycle 


- Lack of SE experience on large complex systems 

- Experienced SE engineers are retiring faster than being trained 


NASA systems engineering for human spaceflight has 
eroded and systems of systems is particularly acute 
(NRC 2004 NASA Systems Integration Study) 

Existing university / industry partnerships are not 
having enough impact 

- SE is not a standard discipline (EE, ChemE, ME etc.) 

- More penetration at undergraduate level 

Need new ways to attract and develop system 
engineers 

- Additional learning 

- On-the-job experience 

- Virtual simulation 




■ 



NRO SE Certification Requirements 


Level 

Experience 

Training 

1 

2 yrs. SE 

SE-501 Acquisition Systems Engineering and 

SE-502 Designing Space Missions 
or 6 SE-related graduate credits 

or SPRDE Level II Certified 

II 

4 yrs. SE 

Complete 4 from below: 

Requirements Development/Management 

Risk Management 
Measurement & Analysis 
Concept & Architecture Development 
Formal Decision Making 

Integration, Verification & Validation 
or 12 SE-related graduate credits or 6 after Level 1 

or SPRDE Level III Certified 

III 

7 yrs. SE 

INCOSE Certification or 

or 18 total SE-related graduate credits or 6 after 
Level 2 




^^duanced P tanning 6 I ntegration ^Jffice 


NASA needs to develop a SE certification program to develop 
systems engineering to meet future program requirements. 





'I 

|l NASA SE Workforce Program 

V 



Establish SE development policy including SE 
certification requirements for promotions 


Establish Government, industry, and academia SE 
education, training, and job experience 
partnerships 


Develop guidelines and process for SE graduated 
certification. Include integration with program 
management education and training 


Measure progress in SE workforce development 
and changes in program SE metrics 
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IA 


Workforce and Education 
Assessment and Vision 



Typical Today 

5-Year Vision 

10-year Vision 

15-Year Vision 

• "erosion of knowledge, 
experience and skills" in "systems 
engineering, project management 
discipline, cost, schedule 
management, and technology 
management", "particularly acute" 
for systems of systems integration. 
(NRC Systems Integration for 
Project Constellation, 2004) 

• DOD has "essentially eliminated 
its systems engineering 
capability". (NRC, 2004) 

• only a single capstone design 
course in undergraduate 
engineering 

• courses taught in traditional 
classrooms 

• some video and Web-based 
Courses 

• A systems engineering 
certification program requiring 
continual education and training 
has been institutionalized 

• just-in-time training via intelligent 
tutoring and advisory systems 

• training support using standard 
NASA and enterprise product and 
process models 

• focused training tuned to new 
opportunities and the best match 
with different employee skills and 
working styles 

• Technological obsolescence of 
workforce virtually eliminated by a 
certified (educated, trained, and 
experienced) systems engineering 
staff for engineering, 
management, and decision 
making 

• learning centers at each of 
NASA’s Collaborative Engineering 
Environment facilities 

• university use of collaborative, 
distributed- learning consortia 

• practical experience of new 
engineers using validated system 
simulations 

• technological obsolescence of 
workforce virtually eliminated 

• Systems Engineering experience 
gained through simulation and on- 
the-job training 

• Advanced Engineering 
Environment technologies and 
systems replicated at the 
university and used for 
maintaining a strong fundamental 
core course structure, with 
simultaneous links to the math and 
science departments and virtual 
links to industry and government 
laboratories 

• national team teaching in 
engineering, math, science, 
management, and the humanities 

• personal learning experience 
emphasized — anytime, anywhere 
via an advanced Internet with high 
bandwidth 

• just-in-time personal/virtual 
training and tutoring 


Adopted from: “Design in the New Millennium: Advanced Engineering Environments”, NRC 2000 
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Systems Engineering 
Tools and Methods 





Target Cost Overrun, Percent 


Effect of Requirements Definition 
Investment on Program Costs 
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The Systems Engineering Process 

(Ref. ANSI 499) 



Process Input 

• Customer Needs/Objectives/ 

Requirements 

- Missions 

- Measures of Effectiveness 

- Environments 

- Constraints 

• Technology Base 

• Output Requirements from Prior 
Development Effort 

• Program Decision Requirements/ 

• Requirements Applied Through 
Specifications and Standards 


n 
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Requirements Analysis 

Analyze Missions & Environments 
Identify Functional Requirements 
Define/Refine Performance & Design 
Constraint Requirement 


System Analysis 
& Control 
(Balance) 


Requirement Loop 

Functional Analysis/Allocation 

• Decompose to Lower-Level Functions 

• Allocate Performance & Other Limiting Requirements to 
All Functional Levels 

• Define/Refine Functional Interfaces (Internal/External) 

• Define/Refine/Integrate Functional Architecture 


Verification 


Trade-Off Studies 
Effectiveness Analysis 
Risk Management 
Configuration Management 
Interface Management 
Performance Measurement 

- SEMS 
-TPM 

- Technical Reviews 


Design Loop 
Synthesis 

• Transform Architectures (Functional to Physical) 

• Define Alternative System Concepts, Configuration 
Items & System Elements 

• Select Preferred Product & Process Solutions 

• Define/Refine Physical Interfaces (Internal/External) 


Systems Analysis and 
Simulation drive the entire 
Systems Engineering Process 


Process Output 

Development Level Dependant 

- Decision Data Base 

- System/Configuration Item 
Architecture 

- Specification & Baseline 
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Integrated Systems Engineering 
and Life-Cycle Management 






Product Life Cycle Engineering 
and Management Focus 






^ Integrated Product Process Development 



IPPD Defined: A management process that integrates all activities from 
product concept through production/field support, using a multi-functional 
team, to simultaneously optimize the product and its manufacturing and 
sustainment processes to meet cost and performance objectives. Its key 
tenets are as follows: 


- Customer Focus 

- Concurrent Development of Products and Processes 

- Early and Continuous Life Cycle Planning 

- Maximize Flexibility for Optimization 

- Use of Contractor Unique Approaches 

- Encourage Robust Design and Improved Process Capability 

- Event Driven Scheduling 

- Multidisciplinary Teamwork 

- Empowerment 

- Seamless Management Tools 

- Proactive Identification and Management of Risk 
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Product Lifecycle Management (PLM) 



On 



Stakeholders 
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Contractors 


Product Life-Cycle Management 

Systems Requirements 
Configuration Items Specifications 
CAD/CAM Standard Database 
Change/Configuration Management 
Virtual/Real System Models 
V/R Production Models 

V/R Verification Requirements and Management 
V/R Validation Requirements and Management 
Comprehensive Production and Quality History 
Resource Management 
Supply Chain Management 




Management 



Engineering 
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Simulation Based Systems Engineering 


Other 

System 



Top Level System 
Requirements 





Cost, Schedule & 
Program Management 







Functional Design 

m 

Distributed 

Simulation 

Framework 


Distributed 

Information 

Repository 


LSJ 

■ 



ystem Info 
depository 


Operations, 
Logistics 
& Training 





Physical & Info 
P System (HW/SW) Design 

. Other 
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r&E 


Engineering 
Development 
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SBSE: The Challenge of 
Contracted Elements 




Cost, Schedule & 
Pr ogram M anagement 



Operations, 
Logistics 
& Training 




Fully Integrate Total NASA/Industry 
Systems Engineering and Management 







Systems Engineering Tools and Gaps 


0 tluancetl P la lining 6 1 ntegration fflce 


Engineering Discipline Tools 

- Mostly very good for detailed analysis; however 
needs standards for multidisciplinary integration 
for design and speed increases for optimization 
and uncertainty analyses. 

Specialty Engineering ("ilities") Tools 

- Little confidence in prediction of causal | 

relationships for reliability, maintainability, 
supportability, operability, availability, safety, etc. 1 

Life Cycle Cost 

- NASA has continually underestimated the life- 1 
cycle cost (technology, development, production,! 
operations, logistics). Needs causal models to 
assist engineering system and lifecycle design. 

Program/Project Management 

- Many excellent tools available for cost, 
schedule, and configuration management; needs 
total integration including risk and engineering 
mitigation planning 

Product Life-cycle Management 

- Many new COTS capabilities are being 
developed. Need to assess and select for NASA 
applications. Integration with simulation based 
SE modeling required. NASA wide and industry 
integration required. 



Critical Gap 1 


Significant Gap 


No or Minor Gap 
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Systems Engineering/Robust Design 




Reliabilit 


aintenance 


Disposa 


Requirements, Flowdown? Trades, 
Sensitivities, and Validation 


Risk Sust 

Cost Inforn 


Informe 
eDecisions 

Safety 


Requirements 
m Concept Development 
w Design/Developm 

Test 


Manufacturing 

Integration/Verifi 


Advanced Tools and Processes 
High Fidelity Numerical Simulatio' 

• Non-Traditional Methods 

• Rapid Synthesis Methods 

• Life Cycle Frameworks 

• Life Cycle Cost Simulations 

• Risk Simulations 


System of Systems 
Life-Cycle Simulation and Modeling 




r ^ 

iS| Integrated Decision-Making 

V 

L A 



Integration of risk analysis with decision processes 


Systems 

Engineering 
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Apollo Decision FOM Matrix (1962) 

V 

L A 




Performance 

Probability of 
Success 

Schedule 

Safety 

R&D Costs 

Ops Costs 

Growth Potential 

Delivery Costs 

Critical Development 
Problem Areas 

EOR 

15300 

14.5 

(w/spare) 

Aug 1969 

18.2 

$6490 E6 

$1240 

12 

$88.4 E6 

a. Earth orbit rendezvous 

b. propellant transfer 

c. C-5 launch vehicle 

d. standard apollo capsule 

LOR 

12,600 

5,000 

LEM 

19.1 

Feb 1969 

16.1 (CM) 
22.0 (LEM) 

$5840 E6 

$620 

10* 

$77.4 E6* 

a. lunar orbit rendezvous 

b. LEM and personnel transfer 

c. C-5 launch vehicle 

d. standard apollo capsule 

C-5 Direct 

9210 

21.9 

Oct 1 968 

16.7 

$5690 E6 

$510 

12 

$61.4 E6 

a. high energy return 

b. light weight capsule 

c. C-5 launch vehicle 

Nova 

Direct 

15300 

25.3 

May 1970 

18.0 

$6160 E6 

$630 

15 

$55.4 E6 

a. Nova launch vehicle 

b. standard apollo capsule 
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Roadmap to Affordability Through 
Robust Design Simulation 



Robust Design Simulation 


Technology 

Infusion 


Physics- 

Based 

Modeling 


Activity and 
Process- 
Based 
Modeling 


Subject to 

Design & Environmental 
Constraints 



rA 

) 

Y 


& Sizing 


® y "'. h f sis J Simulation l Operational _ Life-Cycle 


Environment 


Economic 


Analysis 


Economic & 


Impact of New 

Discipline 


Technologies- 

Uncertainties 


Performance & 



Schedule Risk 


Robust Solutions 

I 1 


Objectives: 

Schedule 
Budget 
J Reduce LCC 
'Increase Affordability 
Increase Safety 
Increase Sustainability 



Customer 

Satisfaction 
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Technology Trends 



MV 



Innovation Focus Throughout 



Optimizing the re-use of Data and Corporate 

Knowledge 
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Systems Engineering 


Product 

Life-Cycle 

Modeling 


Design & 
Manufacturing 


Knowledge 
Capture and 
Management 


Integrated Virtual 
and Real Design, 
Test, Production, 
and Operations 


VPM 


PPR 


Knowledge 

Inside 


3D Collaboration Tools 




1970 1 98 > 1995 2000 2005 2010 







^ Rapid (Virtual and Real) Prototyping 



• Early Requirements Development 

• Analysis of Alternatives 

• Reconfigurable Designs 

• Real/Virtual Integration 

• Human/Machine Performance 



Space 

Transportation 
Systems 
(moon & Mars) 


Constellation System of Systems 


Crew Transport, Launch Systems & Ground Systems 


And Beyond 


In-Space Support 
Systems 
(e.g. Communication 
System, EVA systems) 

... m xi 


/ Planetary 
Access 

(moon & Mars) 


. 


/ V Mars 


Moon / 


Destination 
Surface Systems 


• Safety, Reliability, Cost Trades 

* Systems of System Integrated Performance and Decision Analysis 


Rapid Validation of Virtual Models for Confident 

Decision Analysis 
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Define, Monitor, and Control the Physical World 



VIRTUAL 


PHYSICAL 




Product & Process 
Knowled 


e 


Production 


INTELLECTUAL 

PROPERTY 


REAL 

OPERATIONS 
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SBSE Integration with Systems Engineering 

V 

L A 




• Design is 
Authored as 
Models 


• Simulation 
Verifies the 
Design 


Validated in 
Simulation 


Progressive Model 
Detail 


Physical Test 
Verifies the 
Simulation 


Complete Product 
Model 


Better Decisions / 
Shorter Development Times 
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Virtual Simulation to Keep and 
Reuse Workforce Knowledge 
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Validated virtual simulation may compensate for lack of 
physical Systems Engineering experience. 
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Simulation Based Modeling 
(SBM) Build Progression 



Zk' 
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Subjective Assessments 

• QFD/AHP 

• System Engineering Tools 



Engineer in the Loop 

• Conceptual/Prelim Engineering 

• Risk - Flight, Development, RMS 

• Cost - Complete Life-Cycle 




Architecture and Technology Trade Studies 
Requirements Definition 


Requirements Validation 
Contractor Source Evaluation 
System Optimization/Robust Design 


Operations Analysis 
Logistic Planning 
Mission Planning 
Flight Readiness 
Validation 


Operator in the Loop 

• Ground Operations 

• Mission Operations 

• Supply Chain Management 


Integration and Testing 


Hardware/Software in the Loop 

• Test Program Def & Refinement 

• Hardware & Software Testing 

• System Integration Modeling 



Pilot in the Loop 

• Ground and Flight Sims 

• Validate Engr Concepts & Techs 


Manufacturer/Tester in the Loop 

• Technology / Producibility Plan 

• Factory Layout / Tooling 

• Virtual/Real Test Integration 



Collaboration/Distributive Environment 


A geographically distributed, integrated, secure, collaborative 
environment which enables life cycle design and analysis capability 
enabling world-class engineering and science applications 
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I'avlu.m Ground 

Processing 0perations 


Communications 
& Tracking 


Risk 
Analysis 


Science Pis 


Mission Analysis 


Mission Analysis 


Science PI 
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Modeling Management Structure 

For STS Logistics, Management and Planning ~70% 




STS Budget "Pyramid" 

(FY 1994 Access to Space Study) 


Generic 

Operations Function 


A/iciklA\ \Mr\i Elem. Receipt & Accept. 
■ip oftZ lcibe^g Landing/Recoveiy 


Veh Assy & Integ 


Launch 


Offline Payload/Crew 


Turnaround 


Vehicle Depot Maint. 


Traffic/Flight Control 


Total 
$M FY94 


19.6 


27.1 


51.5 


75.9 


112.3 


237.5 


199.4 


pport (Hidden) 

Recurring 

Ops 


Total 


o.o% 


0 . 6 % 


0 . 8 % 


1.5% 


2.3% 


3.3% 


7.1% 


5.9% 


Operations Support Infra 

318.6 

9.5% 

Concept-Uniq Logistics 

842.7 

25 . 1 % 

STS Ops Plan'g & Mgmnt 

1477.4 

43 . 9 % 

Total ($M FY94) 1 3363.4| 

100.0% 

Percent 

ioo.o%| 



- 10 % 


- 20 % 

- 70 % 



CM McCleskey/NASA KSC 
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Simulation 


Organizational 



* Management and Organization integration is a major 
percentage of program costs 


Information flow, decision paths, and process 
graphs can be stochastically modeled for duration, 
human capital, and impact on total program costs. 


* Currently, no organizational model has been 
developed to analyze NASA program organizational 
performance. 


• Validated organizational simulations may have as 
much impact as system simulation and optimization 
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Steps in the Design and 
Development of Products 
and Processes 

Typical Today 

5-Year Vision 

10-year Vision 

15-Year Vision 

1. Mission Requirements 

• traditional systems 

• establishment of NASA-wide 

• integrated systems 

• all life-cycle engineering 

Analysis/Product System 

engineering methods / non- 

policy and guidelines for 

engineering and management 

functions are seamlessly 

Strategy 

• high-level systems engineering 

standard application across 
NASA 

systems engineering 
• integrated life-cycle analysis 

systems for technical and 
programmatic risk 

integrated for system design, 
development, manufacture, and 
operation 

analysis 

• little integration and reuse of 

tools for system and 

• validated life-cycle simulation 


• stakeholder/mission 

engineering analyses 

requirements trades for 
acquisition 

of all mission requirements 

• all mission and enterprise 
requirements can be traded 

requirements definition 

• late trades of requirements 
versus system specs, 
performance, and cost 


• seamless transitioning of 
technical simulations to 
management and control 
simulation 

• systems of systems 
requirements are understood 
and validated 

with functional and physical 
models for the systems of 
systems environment 

• complete emersion of 
stakeholder in the 
design/requirements process 

2. Product Specification 

• competitive comparisons 

• complete linkage of customer 
requirements, functional 

• knowledge base for 
construction of systems 

• reliable “batch of one” 
methods for unique products 

• product strategy 

• voice of the customer 

• projections of future products 

• interviews and focus groups of 
customers and others 

requirements, physical 
architecture, and operational 
requirements 

analyses for a proposal with a 
"selected" level risk 

• reliable specifications even for 

• product created on demand 

• ability to write in preferences 

• environmental and other 
regulatory requirements 

• demonstrations 

• virtual prototypes for 
specification validation 

first-of-a-kind products 
• systems of systems impact of 

and requests 

• maximum reuse of hardware, 

• planned product specification 

• output is written 
documentation 

• strategic decision models and 
analyses based on uncertainty 
and risk 

• product life-cycle model for 
management of complete digital 
product database 

specifications are known 

software, infrastructure, and 
knowledge for the enterprise 



Adopted from: “Design in the New Millennium: Advanced Engineering Environments”, NRC 2000 




Systems Engineering Tools and Methods 
Assessment and Vision 



IA 



Steps in the Design and 
Development of Products 
and Processes 

Typical Today 

5-Year Vision 

10-year Vision 

15-Year Vision 

3. Concept Development 

• target setting 

• brainstorming on product 
and process alternatives 

• development of product 
and process concepts 

• iterative, largely manual, 
bottom-up, non-optimized 

• expert opinion for concept 
initiation 

• rules of thumb 

• innovation relies on 
experienced practitioners 

• integrated, predictive life- 
cycle cost and profitability 
models 

• optimization of shared 
resources 

• better models of cost and 
"ilities" for concept trades 
with customer requirements 

• complete life-cycle 
optimizations trading 
safety, performance, life- 
cycle cost, 

technical/performance risk, 
and schedule 

• full automation of 
subsystem and component 
tracking and trade-offs 

• collaborative engineering 
environment for complete 
enterprise participation in 
engineering and 
management with 
contractors 

• virtual prototyping for 
manufacturing, integration, 
testing, ground and fight 
operations 

Steps 3, 4, and 5 
combined 

• concept is optimized to 
meet mission and 
enterprise requirements 
(hardware, software, and 
knowledge reuse known) 

- sensitivities, robustness, 
uncertainties are 
automatically generated for 
decision analysis 

• expert system generates 
alternatives 

• optimized, top-down 
concept development 
process 

• automatic analytical 
evaluation of all product 
and process attributes 
(including risk and 
uncertainty) 

• global collaborative 
engineering environment 


Adopted from: “Design in the New Millennium: Advanced Engineering Environments”, NRC 2000 
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Systems Engineering Tools and Methods 
Assessment and Vision 



Steps in the Design and 
Development of Products 
and Processes 

Typical Today 

5-Year Vision 

10-year Vision 

15-Year Vision 

4. Preliminary Product 
and Process Design 

• high-level definition of 
product and process 
designs 

• evaluation of product and 
process designs vs. targets 

• high-level system trade- 
offs 

• iterative, largely manual, 
largely bottom-up, heuristic 

• derivations of existing 
designs 

• progressive definition 

• coarse definition, mostly 
manual from scratch 

• unequal levels of 
definition for new and 
reused parts 

• 20% of product and 
process attributes 
evaluated analytically using 
simplified models 

• reliance on physical 
prototypes 

• rapid iteration of product 
and process design 

• object-oriented models 
scalable from macro to 
micro levels 

• single interoperable data 
set 

• automated process model 
creation 

• analytical evaluation of all 
attributes, including cost 
and producibility 

• multifunctional 
optimization 

• some degree of iteration 
implied, but guided by 
optimization capability 

• analytical evaluation of all 
attributes, 200 to 300 times 
faster than current methods 

• integrated; single data 
source 

• full automation of 
subsystem and component 
tracking and trade-offs 

• virtual manufacturing 

• single-pass product and 
process design and 
concurrent evaluation with 
multifunction optimization 
and automatic cascade to 
next lower level of design 

• automated generation of 
details about component 
and subsystem design and 
manufacturing details from 
high-level descriptions and 
desired attributes 

• single product life-cycle 
data source 


Adopted from: “Design in the New Millennium: Advanced Engineering Environments”, NRC 2000 
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Systems Engineering Tools and Methods 
Assessment and Vision 



Steps in the Design and 
Development of Products 
and Processes 

Typical Today 

5-Year Vision 

10-year Vision 

15-Year Vision 

5. Refinement and 

• detailed process and 

• distributed, collaborative 

• automatic configuration 

• automatic verification of 

Verification of Detailed 

product definition mostly 

processes within NASA 

control and tracking of 

the system and processes 

Product and Process 
Designs 

manual and from scratch 

• physical prototypes 

system and processes 

generated within the NASA 
advanced engineering 

• development of designs 

• limited reuse of design 
geometries for new parts 

essentially eliminated 

• distributed, collaborative 
processes (NASA and 

environment 

for components, 


• real-time sharing of 

contractors) 

• immersive design and 

subsystems, and 

• analytical evaluation of 

design information 


evaluation environment 

manufacturing processes 

one-third of product and 
process attributes using 


• design advisors 

from the total NASA/ 
contractor engineers, 

• geometry creation 

• prediction and evaluation 
of all product and process 
attributes 

• tracking and trade-offs of 
subsystems and 
components 

detailed models 

• some model sharing 

• reliance on physical 
prototypes 

• attribute prediction and 
evaluation partially 
automated, but not 
integrated with design 
evolution 


• minimal, “surgical” testing 

• no late trade-offs and no 
errors 

managers, and decision 
makers 

• international distributed, 
collaborative processes 


Adopted from: “Design in the New Millennium: Advanced Engineering Environments”, NRC 2000 
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Systems Engineering Tools and Methods 
Assessment and Vision 



Steps in the Design and 
Development of Products 
and Processes 

Typical Today 

5-Year Vision 

10-year Vision 

15-Year Vision 

6. System Prototype 
Development 

• experimental refinement of 
product attributes that do not 
meet targets 

• analytical evaluation required 
for more than half of all product 
attributes 

• real and virtual prototypes 
available for form, fit, and 
function demonstrations and 
tests 

• integrated database for 
development of rapid 
prototypes 

• virtual prototypes becoming 
the norm for NASA 

• complete virtual prototyping of 
system, systems, 
manufacturing, integration, 
tests, and operations 

• validated virtual models - 
limited experiments required 

7. Production, Testing, 
Certification, and Delivery 

• virtual shop floor modeled 

• discrete event optimized 
production flow 

• product life-cycle model used 
to integrate production with 
resources, supply chain, 
workforce, and management 

• all production hardware, 
software, infrastructure, 
workforce, and processes 
developed and tested virtually 

• complete integrated virtual 
environment for supply chain, 
production, integration, 
verification, and validation 


• on-line statistical process 
control 

• products with 100% 
quality — getting it right the first 
time 

• complete supply chain 
modeled and integrated with 
production 

• virtual design and 
manufacturing process with 
zero defects 




• off-line robust design 

• lean, agile manufacturing 

• only minor facility 
reconfigurations required for 
single product runs 




• design for manufacturing: 
fewer parts, more compatibility, 
and easier assembly processes 



Adopted from: “Design in the New Millennium: Advanced Engineering Environments”, NRC 2000 



Systems Engineering Tools and Methods 
Assessment and Vision 



IA 



Steps in the Design and 
Development of Products 

Typical Today 

5-Year Vision 

10-year Vision 

15-Year Vision 

and Processes 





8. Operation, Support, 

• sequential, historically based 

• consideration of 

• autonomous systems 

• autonomous systems 

Decommissioning, and 
Disposal 

modeling approach 

remanufacturing in design 

• operations driven supply chain 

• self-healing 


• a lot of manual operations 

• limited autonomous systems 

fully modeled and managed 

• self-disassembly 



• simulation models based on 

• design for easy repair 




operational processes 

• design for disassembly 

• self-disposal 



• improved automation of 
support activities 

• design for reuse and 
remanufacture 




• supply chain modeled for 
impacts on design 




Adopted from: “Design in the New Millennium: Advanced Engineering Environments”, NRC 2000 



Capability 15.1 Systems Engineering 

Roadmap 


Key Assumptions: 
Exploration & 

Science 



Capability Roadmap 
15: Systems 
Engineering Risk/Cost 
Ana l ys i s 


2008 CEV 
Initial Flight 

► 


4 


Initial 

Life-Cycle Management 
Capability 


2011 James 
Webb 


2013 Comet Surface 
Sample Return 

— - — ► 


2015 Prepare for 
Lunar Missions 

A — ► 


4 


Integrated System 
Engineering and 
Management Capability 


4 



Collaborative/Distributive PLEM 
Simulation-Based Capability 


15.1 Systems Engineering 


Initial SE 
Implementation 


Validated, Integrated 
SE Environment 


Collaborative/Distributive PLEM 
ASimulation-Based Capability 


15.1.1 Processes 


15.1.2 Skills 


15.1.3 Tools and Methods 


NASA HQ 
SE Policy 


NASA HQ SI : Standard With Integrated SE, PM, & 


Systems of : 


ystems 


RM (C 


CM I Level 3) 


NASA HQ SE 
Guidelines 


SE Certification Pol icy 

& Program i initial Qertifiec 


Class 


Integratec 


Initial PLJVL 
Implementation 


CM Ml Level 5 
Initial Skilled SE 


Architects 

PLM 



National 
NASA/Industry 
SE Integration 

Distributed SE’s 
Throughout NASA 

Integrated PLM 


Hardware in Loop 7 
3M Build 4 


SBM Build 1 


SBM Build 2 


Systems of Systems 
SBM Build 3 


Pilot in Loop SBM 
Build 5 


2005 


2010 


2015 


Major Decision 

Major Event / Accomplishment / Milestone 
4 Ready to Use 


Legend 

PLM - Product Life Cycle Management 
SBM - Simulation Based Modeling 
CMMI - Capability Maturity Model Integration 
QRA - Quantitative Risk Assessment 
LCC - Life Cycle Cost 
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Capability 15.1 Systems Engineering Roadmap 


Key Assumptions: 2018 Deep Drill & Completed 

Exploration & Initial Human Landing 

Science 


2025 Extended Lunar Capability 
& Life Finder Telescope 



Capability Roa 
15: Systems 
Engineering Risl</Cost 
A n a lysis 




International Collaborate 
Engineering / Management 
Simulation Based Capabilit 


15.1 Systems Engineering 


Initial International Collaborative Engineering / 
Management Simulation Based Capability 


15.1.1 Processes 


International 

SE 



\ Standards 


15.1.2 Skills 



International C 
SE’s 

15.1.3 Tools and Methods 

] 





ertified 


International Collaborative 
Total PLM Environment 


Validated SE L5 PLM 
Environment 


2020 


2025 


2030 


♦ 


▲ 

t 


Major Decision 

Major Event / Accomplishment / Milestone 
Ready to Use 


Legend 

PLM - Product Life Cycle Management 
SBM - Simulation Based Modeling 
CMMI - Capability Maturity Model Integration 
QRA - Quantitative Risk Assessment 
LCC - Life Cycle Cost 
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Summary 



* Systems Engineering in NASA needs to be 
improved for large complex systems of systems 
projects 


* Standard system engineering policy needs to be 
developed at the Agency level for guidance to 
Centers 


* The training and education of systems 
engineering needs to be institutionalized 

• Advanced Engineering Environment can greatly 
enhance program execution, workforce training, 
and search for innovation and improved science 
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Capability - 15.2 Life Cycle Cost 


Presenter: 

Dr. David Bearden 
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■ 


What is a Life Cycle Cost 




• An integrated, process-centered, and disciplined approach 
to life cycle management of projects provides real and 
tangible benefits to all project stakeholders. 

• A LCC estimate includes total cost of ownership over the 
system life cycle, all project feasibility, project definition, 
system definition, preliminary and final design, fabrication 
and integration, deployment, operations and disposal 
efforts. 


• A LCC estimate provides an exhaustive and structured 
accounting of all resources necessary to identify all cost 
elements including development, deployment, operation and 
support and disposal costs. 


* Definitions provided by the NASA Cost Estimation Handbook, 2004 
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Benefits of the Life Cycle Cost 



* “Ensure cost realism and accuracy” 

- The President’s Commission 

* Improve confidence in selection process 

- Enables better budgeting 

* Predict cost impact of change 

* Limit potential for significant overruns 

- Increases mission success 

* Gauge economic impact of decisions 
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Cost T earn Process 



• Evaluated current Capability Readiness Level (CRL) of cost 
discipline, at the lowest cost team WBS level 

- Cost Analysts at NASA HQ, MSFC, JPL, SAIC and The 
Aerospace Corporation evaluated the readiness level and 
importance of the current State of the Practice 

- Scored Robotic Spacecraft and Human Space Flight separately 


* Interviewed Agency cost estimating leaders for current 
status / initiatives 


• Identified remaining near-term gaps after implementation of 
current initiatives 

- Recommended additional measures for near-term 


Envisioned ideal state for cost estimating 
- Five and twenty year horizons 
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IA 


Current State-of-the-Practice for 
Life Cycle Cost 



Tools 

- Primarily system level parametric 
models with broad application 

- Medium fidelity models for 
development and operations 

- Low fidelity requirements (Physics) 
based models for instruments 

- High fidelity component models 
limited in application 

- Immature technology development 
capability 

- Scattered, sparsely-populated 
databases deployed across centers 
and industry 

- Databases with limited content, pre 
full-cost accounting and not 
normalized 


Skills 

- Limited formal cost training in 
academia 

- Limited career path 

Process 

- Program costs rolled up from 
several models 

- Costs validated through 
comparison of bottom’s up to 
parametric (top down) 

- Periodic intersection of cost 
estimation with project 
development 

- Immature linkage to Schedule 
Analysis 

- Minimal understanding of 
relationship of LCC to mission risk 
and safety 
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Maturity Level - State of the Practice for 15.2 Life 

Cycle Cost 






Robototic Spacecraft 

Estimate Life Cycle Cost 

Tools 

Skills 

Process 

Technology Maturation I 




Development 




Production 




Operations 





Human Spaceflight 

Estimate Life Cycle Cost 

Tools 

[Skills 

Process 

Technology Maturation 




Development 




Production 




Operations 





Critical Gap 1 


Significant Gap 


No or Minor Gap 



Results indicate a strong need for Technology Maturation Cost Estimation Capabilities 








Observations on Maturity 



• Capability ratings trended higher for Robotic Spacecraft than 
Human Spaceflight primarily because of better data 
availability (function of more recent, relevant missions) 

• Capability ratings for Technology maturation cost estimating 
low in all areas 

• Production and Development estimating limited by data 
available in Human Spaceflight area 

• Operations cost estimating readiness low due to less mature 
tools and processes and availability of fewer estimators 
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IA 


Requirements/Assumptions 
for Life Cycle Cost 



• Missions Driving Requirements 

- Primarily driven by ESMD 

• Prometheus 

• Crew Exploration Vehicle 

• Human Exploration of Moon/Mars 

- Large SMD Projects 

• James Webb Space Telescope 

- Scale of large ESMD and SMD projects increases budgetary 
impact of overruns, poor estimation, and requirements creep 

• Additional reports that drive capability 

- 2004 Aldridge Commission Recommendations On NASA Cost 
Estimating 

- 2004 GAO Report on NASA Cost Estimating 

- NPR 7120.5C 

- 2004 NASA Cost Estimating Handbook 


82 







Elements of LCC Roadmap 



• Tools 

- One NASA Cost Engineering (ONCE) Database 

- Technology Development Estimation Capability 

- Integrated Cost, Risk, & Schedule Models 

- Integrated Life Cycle Models with Improved Operations Models 

- Requirements (Physics) based Models 

- Economic Modeling 

• Skills 

- Continuous Development 

- Formal Academic Education 

• Process 

- CADRe (Cost Analysis Data Requirement) feeds data to ONCE 

- CCRM (Continuous Cost Risk Management) 

- Standard WBS 

- CAIG-like (Cost Analysis Improvement Group) implementation 
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Cost Estimating 5 Year Vision 



“Enable a more agile cost estimating capability that interacts 
effectively with the project management function” 


• Improved models 

-Representative Initiative: Integrated Life Cycle parametric system level 
models 


-Remaining Gap: Importance of accurate cost information justifies more 
investment to build higher fidelity integrated models 


• Improved database 

-Representative Initiative: CADRe -> ONCE 

-Remaining Gap: Better coordination and cooperation by data owners 
(data sharing by centers/ involved parties), data availability is a long- 
term problem 

• Enhanced process to enable use of LCC estimating as an input to the 
project management function 

-Representative Initiative: CCRM 

-Remaining Gap: CCRM implementation will be challenging 
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Major Decision 


2005 


2010 


A Major Event / Accomplishment / 
Milestone 


t 


2015 


Ready to Use 










Cost Estimating 20 Year Vision 



“Create a cost estimating capability that simulates the 
economic system and interacts seamlessly with management 
and systems engineering throughout the project” 


• Understand the whole economic system and simulate to 
understand the effects of design and programmatic decisions 
have at the industry base level 

- Model not only design solution, but economic business case for 
industry 


• Link the project management and systems engineering process 
with cost analysis 

- Simulate technology changes, process changes, etc. 


Improve tools and databases to allow for high-fidelity analysis 
- Cost as a function of safety, risk, schedule, and technology 
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Capability 15.2 Life Cycle Cost Roadmap 


Key Assumptions: 2018 Deep Drill & Completed 


2025 Extended Lunar Capability 



2030 Prepare for 


Science ^ 

; ^ ,ma„ ,v,a,o 



Capability Roadmap 
. 15: Systems 

Engineering Risk/Cost 
Analysis 

International Collaborative^^ 
Simulation Based Capability^ — -- 

niioijrara 


15.2 Life Cycle cost i 

Decisions based on 
Economic LCC 

k Models i 

LCC imbedded in all X \ 

Agency Decisions 

15.2.1 Tools Closed Economic ) 

^ 4 4 Linked LCC Models for all , 

k Open Economic 

based LCC m 

models 

A 

L i \ phases of project ^ 

Hiaher Fidelity Databases Available 

^ Z 

A based LCC models 

15.2.2 Skills 

LCC Skills readily 
available A 



15.2.3 Process 7 

Continuous cost risk 

\ analysis broadly used / 

\ LCC used for all 

Z * within agency z — x Agency decisions 
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Life Cycle Cost Goals 


Cost Accuracy 


Schedule Accuracy 


DATABASE 


% of Programs w/ 
Complete CADRe 


SKILLS 


% Staff w/ Formal 
Training within 
NASA 


PROCESS 


% Programs 
implementing full 
CCRM process 


30 % 


30 % 


50 % 


50 % 


30 % 


20 % 


20 % 


90 % 


75 % 


60 % 


10% 


10% 


100 % 


90 % 


90 % 



Ei* 

^^duanced P lanning 6 [Integration ^)ffice 
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Evaluated current capability of cost estimation 
discipline 

Envisioned ideal future state for cost estimating 

Performed gap analysis taking into account 
current initiatives 

Developed roadmap from current state-of-practice 
to envisioned state 





Capability - 15.3 Risk Management 


Presenter: 
Theodore Hammer 








Capability - Risk Management 



• Risk Management identifies potential problem areas 
early enough to allow development and implementation 
of mitigation strategies. This includes contingency 
planning, descope approaches, and qualitative and 
quantitative assessments. As complexity of systems 
grows the importance of risk analysis increases in 
managing cost, schedule and mission success. 


• The Risk Management sub-element needs to be 
thoroughly integrated with other aspects of systems 
engineering 


• Risk management includes tools, processes, and skills 
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Key Points/Benefits 



Iduanced I tanning 6 Integration Ufflte 


Risk Management most effective when integrated with 
program/project and technical management 


• Gaps exist within the present risk management state of the 
practice 


* First End State targets elimination of existing gaps 


• End States target delivery of capabilities five years prior to a 
milestone 


* Regular evaluation critical 

• A formal integrated risk management capability benefits 
implementation of highly complex systems by 

- Enabling cost effective implementation and problem avoidance 

- Increasing probability of mission success 

- Reducing programmatic problems (e.g., cost and schedule) 
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IA 


Current State-of-the-Practice for 
Risk Management Within NASA 



• Risk Management policy and requirements exist 

• Conduct annual NASA Risk Management conference 

• Risk Management planning widely used 

• Assessments are highly qualitative 

• Quantitative assessments using such tools as PRA are limited 

• Risk mitigation planning and implementation widely used, but not well 
integrated into the project planning (e.g., cost/work breakdown, 
integrated schedules) 

• Various risk management tools have been used, however , based on 
NASA trade studies ESMD has selected a state-of-the-art risk tool as 
the Directorate standard: Active Risk Manager (Strategic Thought, LLP) 

• Formal risk management training exists based on Software Engineering 
Institute risk management process 


Evaluation based on OSMA and NASA Center RM POC assessments. 
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Evaluation of Risk Management 
State of the Practice 



IA 


Risk Management 



Skill 


Tool 


Process 


Prepare for Risk Management 





Determine Risk Sources and Categories 




Define Risk Parameters 




Establish a Risk Management Strategy 









Identify and Analyze Risks 





| Identify Risks 





Quantitative 




Qualitative 




Evaluate, Categorize, and Prioritize Risks 





Planning 





T rack/Control/Communicate 









Mitigate Risks 





Develop Risk Mitigation Plans 




Implement Risk Mitigation Plans 





Critical Gap 


Significant Gap 
No or Minor Gap 
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■ 



• Prepare R 

- Insufficient level of integration of risk management and risk 
assessment with other capabilities 

- Lack of regular collection of data to assess the level of compliance 
and practice of risk management and assessment 

- Limited skill, tools and process for in-depth identification of risk 
sources 

- Limited skill, tools and process for an integrated risk strategy 

• Identify R 

- Lack of standardization in risk management tools used 

- Inconsistent level of skill and knowledge for Risk Management 
practioners 

- Insufficient application of quantitative techniques to identify risks, and 
limited qualitative assessment skills 

- Insufficient skills and tools for a consistent approach to monitoring, 
tracking, control/feedback and communication (e.g., external) of risks 

• Mitigate Y 

- Limited skill and tools for mitigation planning 

- Limited skill, tools and process for the implementation of mitigation 
activities 
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IA 


Requirements/Assumptions 
for 15.3 Risk Management 



• Key Assumption is capability to support key milestones must be in 
place 5 years prior: 

- 201 1 James Webb Telescope 

- 2015 Prepare for Human Lunar Missions 

- 2018 Initial Human Lunar Landings 

- 2025 Extended Lunar Capability 

- 2030 Prepare for Human Mars Mission 

• Requirements and assumptions for increased risk management 
capabilities 

- Increased complexity of systems 

- Increased inter-dependency of complex systems 

- Distributed implementing organizations 

- Environment uncertainty 

- Longer mission durations/complex logistics requirements 

- Tougher science requirements 

- Challenge of implementation and verification of advanced instrument 
technology (e.g., increased detector sensitivity) 

- Increase future IT capabilities at lower costs 
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FY 2010 Lunar Support 


End States 



* Prepare 

- Change process and skills to effect integration of risk management 

- Regular collection of self assessment data 

- Institute skills, tools and process for: 

• In-depth identification of risk sources 

• Integrated risk strategies 

* Identify 

- Standardize risk management tools used 

- Define skills/knowledge criteria for risk practioners; conduct training 

• Including quantitative techniques 

- Institute skills, tools: Monitoring, tracking, control/feedback and 
communication (e.g., external) of risks 

* Mitigate 

- Institute skill and tools for mitigation planning 

- Institute skill, tools and process for the implementation of mitigation 
activities 
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IA 


Top Level Objective of RM 2009 

End State 



Integration of risk analysis with decision processes 
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End States (Continued) 


FY 2014 Human Lunar Landing Support 



• Prepare 

- Improved risk source identification; expanded to include 
routine operational environment challenges 

- Risk sensitivity analysis for interdependent complex systems 


* Identify 

- Simulation-based risk identification 

- Increased depth and fidelity of quantitative techniques 

- Improved risk communication, including risk uncertainties 


Mitigate 

- Integration of mitigation activities into project schedules 


99 








End States (Continued) 



FY 2020 Extended Lunar Support 

* Prepare 

- Risk sensitivity analysis techniques for interdependent 
systems 

- Improved risk source identification; plans for expanded 
extended lunar operational environment challenges 

* Identify 

- Predictive risk capability and tools 

- Interactive risk identification; knowledge based providing a 
connection to risk decisions made in the past 

* Mitigate 

- Capture of risk mitigation successes/failures to predict 
mitigation approach probability 
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End States (Continued) 


FY 2025 Human Mars Support 



• Prepare 

- Improved risk sensitivity analysis techniques for 
interdependent complex systems 

- Improved risk source identification; plans for 
expanded Mars operational environment 
challenges 

* Identify 

- Improved predictive risk capability and tools 
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Capability 1 5.3 Risk Management Roadmap 



Key Assumptions: 
Exploration & 
Science 



Capability Roadmap 
15: Systems 
Engineering Risk/Cost 


2008 CEV 
Initial Flight 

► 


2011 James 2013 Comet Surface 2015 Prepare for 
Webb Sample Return Lunar Missions 


Analysis 




Initial 

Life-Cycle Management 
Capability 


i 


Integrated System 
Engineering and 
Management Capability 



Collaborative/Distributive 
PLEM Simulation-Based 
^Capability 


15.3 Risk Management 


Integrated Risk Tools 
a and Mitigation plans 


15.3.1 Prepare 


15.3.2 Identify 


15.3.3 Mitigate 


Integrated Risk 
Process 


Risk ID Tools 
Developed 


Risk Mitigation 
Plans Routinely 
Used 


Data base of Self 
Assessment Established 


Standardized Risk 
Tools Used 


Standardized Simulation 
Based Risk Analysis 


Generic Risk Strategies 
Data Base Developed 


Simulation Based 
Risk Identification 


Risk Mitigations Tracked 
against Identified Risks 


Probabilistic Risk 
Analysis and Mitigaji 

Operational Environment 
included in Risk A nalysis 


Probabilistic/Sensitivity 
Analysis Risk Identification 


Risk Mitigations Integrated 
into Project Schedules 


2005 


2010 


2015 


f 


Major Decision 

Major Event / Accomplishment / Milestone 
Ready to Use 


Legend 

PLM - Product LC Management 

SBM - Sim. Based Modeling 

CMMI - Capability Maturity Model Integration 

QRA - Quantitative Risk Assessment 
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Capability 15.3 Risk Management Roadmap 

V 

L A 



Key Assumptions: 2018 Deep Drill & Completed 

Exploration & lnitial Human Landing 

Science * 


2025 Extended Lunar 
Capability & Life Finder 

"bji T ele s CO pe 





2030 Prepare for Human 
Mars Mission 




Capability Roadmap 
15: Systems 
Engineering Risk/Co^t 
Analysis 


I 


Initial International Collaborative Engineering I International Collaborative 
/ Management Simulation Based Capability Engineering / Management 

Simulation Based Capability 


15.3 Risk Management 


15.3. Interdependent 
Risk Analysis 


15.3 Interactive Risk 
^ Identification and Mitigation 


15.3 Accurate Risk Analys 
Uncertain Environments 


15.3.1 Prepare 


15.3.2 Identify 


15.3.3 Mitigate 


Interdependent Systems 
Risk an alysis 


Extra-terrestrial 
operational environments 
included in risk analysis 


Interactive Risk 
Identification 


Predicti /e tools/Processes 

Generic Risk 
Mitigations data base 
developed 


Interdependent System of 
Systems Risk analysis 


Project-based Real- 
time risk identification 
and mitigation 


Interactive Risk 
Identification and 
Mitigation 


2020 


2025 


Accurate Risk 
Analysis in Uncertain 
Environments 


2030 


f 


Major Decision 

Major Event / Accomplishment / Milestone 
Ready to Use 


Legend 

PLM - Product LC Management 

SBM - Sim. Based Modeling 

CMMI - Capability Maturity Model Integration 

QRA - Quantitative Risk Assessment 
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Maturity Goals 



RISK MANAGEMENT 



fldiianced P tanning 6 

1 ntegration ^Jffi 


2009 

2015 

2020 

2025 

Prepare for Risk Management 

Change process and skills to effect integration of RM 

6 

7 

7 

7 

Regular collection of self assessment data 

1/YR 

1/YR 

1/YR 

1/YR 

Institute skills, tools and process 

80% 

100% 

100% 

100% 

Improved risk source identification 


6 

7 

7 

Risk sensitivity analysis for interdependent complex systems 


6 

7 

7 

Sensitivity analysis techniques for interdependent complex systems 



6 

7 

Improved risk source id; extended lunar operations 



6 

7 

Improved risk source identification; expanded Mars ops 




6 

Identify and Analyze Risks 

Standardize risk management tools used 

6 

7 

7 

7 

Define skills/knowledge criteria for risk practioners 

6 

7 

7 

7 

Institute skills, tools: Monitoring, tracking, control/feedback and communication 

6 

7 

7 

7 

Simulation-based risk identification 


6 

7 

7 

Increased depth and fidelity of quantitative techniques 


6 

7 

7 

Improved risk communication, including risk uncertainties 


6 

7 

7 

Predictive risk capability and tools 



6 

7 

Interactive risk identification; knowledge based connection to risk decisions made in the past 


6 

7 

Improved predictive risk capability and tools 




6 

Mitigate Risks 

Institute skills and tools for mitigation planning 

6 

7 

7 

7 

Institute skill, tools and process for the implementation of mitigation activities 


6 

7 

7 

Integration of mitigation activities into project schedules 


6 

7 

7 

Capture of risk mitigation successes/failures to predict mitigation approach probability 



6 

7 
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Summary 






• Risk Management most effective when integrated 
with program/project and technical management 


* First End State targets achieving RM integration 
with program/project and technical management, 
and elimination of existing gaps 


* End States target delivery of capabilities five 
years prior to milestone that would benefit most 
from those capabilities 


* Regular evaluation critical to determining 
capability maturity and success in meeting end 
state objectives 
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Capability - 15.4 Safety & 
Reliability Analysis 

Presenter: 

Homayoon Dezfuli, Ph.D, NASA 
Team Lead 
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IA 


Objectives of System Safety & 
Reliability Analysis 



• Evaluation and management of 

- Safety risk 

- Mission success 


• Includes processes and techniques used to provide organized, 
disciplined approach to: 

- Identify and resolve risks as effectively as possible 

• Personnel 

• Equipment 

• Mission success 

- Assess safety and reliability through all phases of the life cycle 

- Risk-informed management of safety & reliability 

• Assessment tools and processes should provide integrated 
evaluation of the entire system: 

- Hardware 

- Software 

- Physical environments 

- Operations 

- Human 

- Interactions of systems 
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IA 


Benefits of 

Safety & Reliability Analysis 



• Benefit: Ensure safety and mission success while affordably 
meeting program objectives 


• This benefit will be realized when safety, reliability and risk 
analyses are standardized and are integrated with decision 
processes under a single decision-making framework 

- Integrate information on safety, reliability and risk under one umbrella 

(integration) 

• Elimination of organizational and process barriers 

- Systematize the hazard identification process (modeling 
standardization) 

- Analyze safety and mission risk (measurement of safety and mission 
performance) 

• Assessment of aggregate risks 

• Identification of weaknesses and vulnerabilities 

• Identification and assessment of uncertainties 

- Manage safety and mission risk (decision-making) 

• Performance of trade-off studies 

• Development of risk reduction strategies 
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IA 


Current State-of-the-practice for 
15.4 Safety & Reliability Analysis 



• Hazard analysis is widely used 


- Focuses on specific contributors 

- Limited applicability to complex systems-of-systems 

• generally the result of brainstorming 


* Fault Tree Analysis and Failure Modes and Effects 
Analysis are widely used 

- Typically applied when completed design information is 
available 

- Primarily applied at subsystem level 

- Limited ability to affect early design decisions 


• Risk Matrix is widely used 

- Applied to top-level risk issues 

- Interaction between risk items is difficult to discern 

- Is unsuitable for combining risks to obtain aggregate risk 

— I InnprtaintiPQ arp nnt fnrmallw arnmintprl fnr 
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^ Example Application of Risk Matrix 



• A Typical State-of-Practice System Safety Assessment 
Technique 

- Analyst postulates a failure or a deviation and assesses its 
consequences 

• Typically one failure or deviation is analyzed at a time 

- Analyst qualitatively judges how often a failure or deviation can 
occur 


- Analyst qualitatively judges the severity of the outcome or assumes 
the worst-case outcome 

- Analyst maps each analyzed failure into one of three risk categories 

( areen, , Red) 



Severity 
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Current State-of-the-practice for 
15.4 Safety & Reliability Analysis (Cont.) 



* The state-of-practice safety analyses does not readily reveal 
whether safety is improving, declining or staying the same 

- Not designed to measure safety 
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IA 


CAIB Report Finding F7.4-4 
(Volume I, page 193) 



“System safety engineering and management is 
separated from mainstream engineering, is not 
vigorous enough to have an impact on system 
design, and is hidden in the other safety disciplines 
at NASA Headquarters.” 
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Current State-of-the-practice for 15.4 
Safety & Reliability Analysis (Continued) 



fl duanced 


P tanning 


* NASA has begun applying probabilistic risk 
assessment (PRA) techniques for evaluating 
safety performance 

- PRA is shown to be an effective tool 

• To integrate qualitative and quantitative safety models 

• To quantify risk metrics relating to the likelihood and severity of 
events adverse to safety or mission success including gaining 
an understanding of uncertainties 


* Probabilistic risk models have not yet been used 
for design decisions 

- Models for software-intensive systems, unique space 
environment, and human decision-making and human- 
automation interactions have not been fully developed 

- Model developments are hampered by lack of PRA skills and 
limited and fragmented safety-related reliability databases 






IA 


Requirements/Assumpt ions for 
15.4 Safety & Reliability Analysis 



* Robust and effective Safety and Reliability Assessment will 
be necessary to safely and affordably meet all the goals in 
the mission framework 

- ~ 14 launches FY05 -FY10 (not including Shuttle and ISS) 

- Over a hundred launches between FY10 - FY 30 

- Planetary missions using nuclear technology 

- Human mission to Mars by 2030 

- Sample & return missions to Mars in 2014 

- Potential for 3 month stay on the Moon 

- Complex science missions (telescopes and solar exploration) 


* Not limited to human safety and crew survival, 

- Must include loss of mission, loss of equipment, and adverse 
environmental impacts 
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IA 


Maturity Level - Capabilities 


for 15.4 Safety & Reliability Analysis 



Skills Tools Processes 


Risk and Safety Management 

Risk Tradeoffs, Risk Acceptance and Risk Communication 

Appreciation and Quantification of Uncertainties 

Mishap Investigation 

Trend and Precursor Analysis 

Dissemination of Lessons Learned 


Systems Safety 

Qualitative Systems Safety Analysis (hardware, software, phenomenological, human) 
Quantitative Systems Safety Analysis (hardware, software, phenomenological, human) 


System Reliability 

Reliability Prediction Models 
Reliability Database 




Minor or No Gap 


Significant Gap 


Critical Gap 

|Text in red indicates a gap 
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Top-level Objective for FY10 
15.4 Safety & Reliability Analysis 



IA 



• Objective: Integration of qualitative and probabilistic 
methods to support design evaluation 

- Integrated qualitative and probabilistic methods are usually not 
conducted until late in the system life-cycle 




State-of- 
practice 
focuses at 
this stage 


System Operation 


Initial 

Design 

Space 


Applying integrated system safety and 
reliability analyses for assessment and trade- 
off studies early in the design process to improve the 

effectiveness of decision-making 
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Top-level Objective for FY10 15.4 
Safety & Reliability Analysis (Continued) 



Integration of risk analysis with decision processes 


Risk-informed Decision-making 

(Integrated Consideration of all Performance Measures and Deliberation) 


TECHNICAL RISK 



Risk Metric 


Risk Metric 

(Loss of 


(Loss of 

Crew) 


Science) 





Performance 
Measures 
(Quantities of 
Interest to 
Decision-Maker) 




Risk Metric 
I (Injury to 
| Public) 


Risk Metric 

Risk Metric 

(Schedule) 

(Cost) 




INTEGRATION OF 
QUALITATIVE AND 
QUANTITATIVE SYSTEM 
SAFETY ANALYSIS 


FM 

EFFECT 

CR 

Device A 
Fails 

Loss of X 

1 

Device B 
Fails 

Loss of Y 

3 


Decision Option 









uv 


Top-level Objective for FY10 15.4 
Safety & Reliability Analysis (Continued) 



Decision 

Choose the most suitable option or 
reduce uncertainty (do more research)^ 
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IA 


FY15 Vision for 15.4 
Safety & Reliability Analysis 



• Safety, consistent with mission requirements, is designed into the 
system in a timely and cost-effective manner 

- Standardization of safety and reliability analyses and processes and their 
integration with systems engineering process 

- Ability to trade safety & reliability against performance, cost, design options, 
diverse management paths 

- Extend analysis philosophy to development stages of system design 

- Developing risk acceptance process and criteria 

- Ability to assess and quantify uncertainties 

- Ability to perform trend and precursor analysis 

- Systems knowledgeable safety experts 


• Physics-based Probabilistic Risk Assessment Models that fully 
integrate all elements of risk; including technical, organizational, 
and cost 

- Centralize existing safety, reliability, system design/operating limitations, and 
risk focused database 

- Assessing expected performance of a design / operational strategy, based 
on probabilistic simufation of time histories and explicit evaluation of 
performance (risk) metrics for those time histories 

- User-friendly, intuitive safety & reliability tool interfaces 

- Risk models linked directly to database with automated evaluation updates 
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Mission Risk 



IA 


Top-level Objective for FY15 
15.4 Safety & Reliability Analysis 





Undesired from Mission Risk Point of View 


Unacceptable Region 



► 


Defining acceptable risk 
regions specific to the 
program 


Risk assessment of decision 
options 


Assessment of uncertainties 


consideration of risk results 
including their uncertainties 
in decision-making 


Safety Risk 
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IA 


FY30 Vision for 1 5.4 
Safety & Reliability Analysis 



• System safety and reliability activities incorporated in a risk- 
informed decision-making framework, capable of 

- Responding to mishaps in real time 

- Allocating resources (presents solutions, evaluates mitigation 
options) " 

- Effective communication of safety issues 

- Monitoring performance using well defined risk metrics 


• Virtual life-cycle simulation model of safety & reliability 

- Next-generation hazard analysis techniques that evaluate 

• New hardware technology 

• Software 

• human performance 

• Organizational factors 

- Safety and reliability models that interface with 

• Quality control processes 

• Testing processes 

• Assembly and manufacturing 

• Maintenance and operational processes 
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IA 


Example of a Simulation-based 

Risk Model 



0 

I 1 1 h 


At 

◄ ► 


H h 


tj = f At 

H 1 h 


Time 

H 1 ► 
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15.4 Safety & Reliability Analysis 


Key Assumptions: 
Exploration & 
Science 



Capability Roadmap 
15: Systems 
Engineering Risk/Cost 


2008 CEV 
Initial Flight 

► 


Analysis 


4 


Initial 

Product Life-Cycle 
Capability 


2011 James 

Webb 




2013 Comet Surface 
Sample Return 

► 


2015 Prepare for 
Lunar Missions 


Integrated System 
Engineering and 
Management Capability 


4 



Collaborative/Distributive 
PLEM Simulation-Based 
Capability 


15.4 Safety & Reliability 


Standard Safety/Reliability 
Management Process with S^=. 
Process 


Advanced Physics-based 
QRA 


Next Generation Hazard 
Analysis Techniques 


15.4.1 System Safety 


15.4.2 System 
Reliability 


15.4.3 Safety 
Management 


Knowledgeable technics 
performing safety analys 


I experts l nte 9 ratec * Hazard Dat ibase 


Model-based Hazard 
Analysis 



Integrated Reliability 
database 


Model-based 
Reliability Analysis 


Integrated Hazard & 
Reliability Model Based 
Analysis 


Complete Set of 
Risk Metrics 


Integrated Assessment and 
Management of Technical 
Risk 


Complete Integration of Risk Analysis with Decision Processes 
(Risk-informed Decision Making) 


2005 


2010 


2015 


4 


Major Decision 

Major Event / Accomplishment / Milestone 
Ready to Use 


Legend 

PLM - Product Life Cycle Management 
SBM - Simulation Based Modeling 
CMMI - Capability Maturity Model Integration 
QRA - Quantitative Risk Assessment 
LCC - Life Cycle Cost 
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15.4 Safety & Reliability Analysis 



Key Assumptions: 
Exploration & 
Science 


2018 Deep Drill & Completed 
Initial Human Landing 



2025 Extended Lunar Capability 
& Life Finder Telescope 


2030 Prepare for 
Human Mars Mission 






Capability Roadmap Initial international Collaborative 

15: Systems Engineering / Management Simulation 

Engineering Risk/Cost Based Capability 

Analysis 


International Collaborative 
' Engineering / Management 
Simulation Based Capability 


~ f . o n ..... Next Generation of Safety & 

15.4 Safety & Reliability - Re li ab ilit y Management System 


Virtual Safety and SE 
Analysis capability 


15.4.1 System Safety 

15.4.2 System Reliability 

15.4.3 Safety Management 


Engine for Integrated Predictive Safety & 
Reliability model-based analysis 



Engine for Integrated Predicfive^E & Safety 
& Reliability model-based analysis" 


Virtual Safety analysis capability 


Integrated Virtual Safety and 
SE analysis capability 


2020 


2025 


2030 


♦ 


▲ 

t 


Major Decision 

Major Event / Accomplishment / Milestone 
Ready to Use 


Legend 

PLM - Product Life Cycle Management 
SBM - Simulation Based Modeling 
CMMI - Capability Maturity Model Integration 
QRA - Quantitative Risk Assessment 
LCC - Life Cycle Cost 
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Concluding Summary 


Presenter: 
Stephen Cavanaugh 









Systems Engineering 


Risk Management 


Life Cycle Costing 



SE-CMMI 


Team 

Assessment 


ENGINEERING 




REQUIREMENTS DEVELOPMENT ! 




REQUIREMENTS MANAGEMENT i 


1 


TECHNICAL SOLUTION 



PRODUCT INTEGRATION 


VERIFICATION 


VALIDATION 


| PROJECT MANAGEMENT 




PROJECT PLANNING ! 




PROJECT MONITORING AND CONTROL ! 




SUPPLIER AGREEMENT MANAGEMENT 




INTEGRATED PROJECT MANAGEMENT FOR IPPD j 




RISK MANAGEMENT j 




INTEGRATED TEAMING ! 


1 


INTEGRATED SUPPLIER MANAGEMENT 



QUANTITATIVE PROJECT MANAGEMENT 


| SUPPORT 



CONFIGURATION MANAGEMENT 


PROCESS AND PRODUCT QUALITY ASSURANCE 


MEASUREMENT AND ANALYSIS 




DECISION ANALYSIS AND RESOLUTION 


1 


ORGANIZATIONAL ENVIRONMENT FOR INTEGRATION 



CAUSAL ANALYSIS AND RESOLUTION 


| PROCESS MANAGEMENT 



ORGANIZATIONAL PROCESS FOCUS 



ORGANIZATIONAL PROCESS DEFINITION 


ORGANIZATIONAL TRAINING 


ORGANIZATIONAL PROCESS PERFORMANCE 


ORGANIZATIONAL INNOVATION AND DEPLOYMENT 



Robototic Spacecraft 

Estimate Life Cycle Cost 

Tools 

| Skills 

| Process 

Technology Maturation 




Development 




Production 




Operations 




Human Spaceflight 

Estimate Life Cycle Cost 

Tools | 

Skills 

Process 

Technology Maturation ! 




Development 




Production 




Operations 






1 Skill | 

] Tool 

Process 

Prepare for Risk Management 





Determine Risk Sources and Categories 




Define Risk Parameters 




Establish a Risk Management Strategy 

1 








Identify and Analyze Risks 




1 

| Identify Risks 





Quantitative 

l 



Qualitative 




| Evaluate, Categorize, and Prioritize Risks 





Planning 





Track/Control/Communicate 









Mitigate Risks 





Develop Risk Mitigation Plans 




Implement Risk Mitigation Plans 





Safety & Reliability Analysis 


Skills Tools Processes 


Risk and Safety Management 

Risk Tradeoffs, Risk Acceptance and Risk Communication 
Appreciation and Quantification of Uncertainties 
Mishap Investigation 
Trend and Precursor Analysis 

Dissemination of Lessons Learned 


Systems Safety 

Qualitative Systems Safety Analysis (hardware, software, phenomenological, human) 
Quantitative Systems Safety Analysis (hardware, software, phenomenological, human) 


System Reliability 

Reliability Prediction Models 
Reliability Database 


Critical Gap 


Significant Gap 


No or Minor Gap 




Key: 


B Minor or No Gap 
Significant Gap 
Critical Gap 
indicates a gs 


indicates a gap 







IA 


Systems Engineering Cost/Risk 
Analysis Roadmap Metrics 



• Development Metrics (process, skills, tools) 

- Annual SE NASA modified CMMI audit of maturity (levels 1-5) 
and capability readiness (levels 1-5) 

- Number of NASA certified engineers in Systems Engineering, 
Life-Cycle Costing, Risk Management, and Safety 

- Percentage of programs using integrated Systems Engineering, 
Project Management, Life-Cycle Costing, Risk Management, and 
Safety tools 

• Performance Metrics (implementation) 

- Number of cancelled programs and termination reviews per year 

- Average percent cost of overrun per year 

- Accuracy of cost and schedule predictions 

- Percent of program cost dedicated to Systems Engineering 

- Number of mission failures per total number of missions 

- Number of hits (requests) from Knowledge Management 
databases in Cost, Reliability, Safety, Risk, and Systems 
Engineering 
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IA 


Systems Engineering Cost/Risk 
Analysis Roadmap Program Review 



• Do the Capability Roadmaps provide a clear path way to 
technology and capability development? 

- Yes. All Roadmap sections address skills, tools (including 
Database creation from which Models are developed to address 
current gaps), and new process. 

• Are technology maturity levels accurately conveyed and 
used? 

- Yes. CRL were assessed by the community, and programs 
created to address areas with low level CRLs. 

• Are proper metrics for measuring advancement of technical 
maturity included? 

- Yes. The development and performance metrics assigned are 
appropriate to measure progress towards increasing the validity 
of the discipline, and reflect current Government criticism. 

• Do the Capability Roadmaps have connection point to each 
other when appropriate? 

- Yes. The capability is a discipline which connects to all other 
roadmaps. 
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IA 


NASA Systems Engineering Cost/Risk 
Analysis Roadmap Team Summary 



• An active Senior Sponsor is absolutely, essential due to the 
complexity of future NASA Exploration missions 

• Develop an Integrated organization of Systems Engineering, Cost, 
Risk, & Safety 

- Application needs to be strategic and tactical implementation 

- Capability to integrate across Agency are currently uneven 

• Develop a Systems Engineering, Cost, Risk and Safety 
Professional Certification program to develop a qualified skill base 

- Require SE certification level for all SE positions 

- Require as a performance objective in personnel reviews 

- Reward progress 

• Establish an independent review process for each program that 
provides a gate keeping processes to ensure project success 

• Create a centralized archival database with best practices, skill 
base, processes, and lessons learned 


The state of systems engineering as practiced at NASA needs to be 
improved to successfully achieve the Exploration Vision. 
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r ^ 

DoD Partnering Possibilities 

V 

L A 



• Both part of the U.S. government with all the general rules, 
regulations and procedures that entails 

• Share a common industrial base 

• Anticipate a large turn over of the workforce in the near future 

• Funding constraints, including uncertainties from budget cuts 

• Moving towards capabilities-based acquisition and evolutionary 
development 

• Increasing complexity with more system-of-systems and families- 
of-systems 

• Share some technology overlap 

• Need a strong role of Systems Engineering Systems Engineering, 
Cost, Risk and Safety within our programs to be successful 


Opportunity exists to collaborate with DoD & NROs Systems 
Engineering Professional Development Program and the established 
Systems Engineering Education programs at DAU & AFIT. 
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Next Steps/Forward Work 

V 

L A 



Make changes to roadmaps based on NRC feedback 

Review and Assess all applicable Strategic Roadmaps and their 
requirements for Systems Engineering capabilities 

- Suggest possible opportunities for Strategic Roadmaps 

Make changes to roadmaps to ensure consistency with Strategic 
Roadmaps requirements 

Additional metrics to determine if achievements will be reached 


Continue to work with other Capability roadmaps to ensure 
consistency and completeness 

Develop rough order of magnitude cost estimates for the Systems 
Engineering, Cost, Risk and Safety Capability Roadmap 

Prepare for 2 nd NRC Review which will address 4 additional 
questions: 

Are there any important gaps in the capability roadmaps as related to the 
strategic roadmap set? 

Do the capability roadmaps articulate a clear sense of priorities among 
various elements? 

Are the capability roadmaps clearly linked to the strategic roadmaps, and 
do the capability roadmaps reflect the priorities set out in the strategic 
roadmaps? 

Is the timing for the availability of a capability synchronized with the 
scheduled need in the associated strategic roadmap? 
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SE Back Up Slides 







Capability Readiness Level Rating 



7 - Commercial processes/tools widely used by industry and NASA 
6 - Commercial processes/tools sparsely used by NASA 

5 - Specialized NASA developed processes/tools used in current 

programs 

3 - Processes/tools under development for existing projects/programs 

1 - Ideas of processes/tools that could enhance NASAs Systems 

Engineering 
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